Method and apparatus for identifying logical volumes in multiple element computer storage domains

ABSTRACT

Method and apparatus for specifying and identifying logic volumes in computer systems that store logical volumes on multiple storage elements are disclosed. The logical volume identifier may be unique with respect to all other logical volumes stored on the storage elements. The logical volumes may be conventional logical volumes, partitions, or hyper volumes.

FIELD OF THE INVENTION

This invention relates to data storage for computers, and moreparticularly to identification of logical volumes stored in multiplestorage elements in a storage domain.

DISCUSSION OF THE RELATED ART

Virtually all computer applications (or programs) rely on storage. Thisstorage can be used for both storing the computer code and for storingdata manipulated by the code. (The term “data” refers to anyinformation, including formatting information, error detection andcorrection information, executable code and data for manipulation by anapplication program.)

Storage technology has developed in a variety of different directions.Accordingly, a wide variety of storage systems are available. It hasbecome impractical, therefore, for the person writing the computerapplication to also be responsible for detailed control over how data isstored on the storage system.

For this (and other) reasons, application programs typically run on anoperating system (e.g., Unix, Windows, MS DOS, Linux, and the manyvariations of each). Once again, however, the operating system may beused with a variety of storage systems.

It would be highly inefficient to have to change the operating system,or the application programs, every time a change is made to physicalstorage. As a result, various layers of abstraction have evolved forviewing how data is actually stored in the storage system.

FIG. 1 illustrates one way of viewing the layers of abstraction. At thetop level 10, the application program may assume that data is stored ina manner that has very little to do with how the data is placed onto thephysical device. For example, the application may view the storagesystem as containing a number of directories and data files within thedirectories. Thus, in an application written for use in the Unixoperating system, the application will assume that files are storedaccording to the Unix directory structure (including hierarchicaldirectories and files located within the directories). This assumedorganization of physical storage may have very little to do with howthat data is actually stored onto the actual storage devices. This viewmay be referred to as the “logical view” because of the separationbetween the logical view of data from the application level is divorcedfrom any view of how the data is physically stored. A logical entity,such as a file, database or other construct, may be referred to at thelogical level as a “logical object.”

The application level 10 interfaces with the file system level 12. Thefile system level is concerned with how files are stored on disks andhow to make everything work efficiently and reliably. Thus, the filesystem level may be responsible for storing directory structure, and forbreaking up files into constituent data blocks for storage onto aphysical storage system. For example, in most implementations of Unix,each file has an associated I-node. This node may contain accounting andprotection information and, additionally, a set of pointers to datablocks.

Relatively early in the development of computer systems, disk drivesbecame a fundamental device for storage. Accordingly, computer operatingsystems have been is developed assuming that memory will rely oninput/output (“I/O”) to a disk drive. The file system 12, therefore, mayassume one or more “volumes” which correspond to a physical storage unitsuch as a disk drive (or any other unit of storage), with data stored inblocks on the disk drive.

The demand for storage to be available for use by applications has skyrocketed. As a result, a number of separate physical devices may berequired to accommodate the total amount of storage required for asystem. In addition, storage systems are often changed or reconfigured.

To insulate the operating system from any changes within the physicaldevice storage system, some mechanism is often employed to flexibly mapa standard (volume) view of physical storage onto an actual physicalstorage system. Thus, an operating system or application may rely on“volumes” of data. Again, these “volumes” may not correspond to how datais actually stored on physical devices. Accordingly, these are “logicalvolumes” (as opposed to actual physical storage volumes) and arethemselves “logical objects.”

The logical volume manager (“LVM”) 14 of FIG. 1 can help achieve thefunction of mapping the file system view of data storage into anintermediate layer.

For purposes of the specification and claims, “logical volume” refers toa logical entity that generally corresponds to a logical abstraction ofphysical storage. “Logical volume” may include, for example, an entitythat is treated (logically) as though it were composed of consecutivelyaddressed blocks in a fixed block architecture or records in acount-key-data architecture. “Logical volume” includes not only standardlogical volumes, but also components of a standard logical volume,hyper-volumes, partitions, striped volumes, and concatenated (or meta)volumes. A logical volume may be physically stored on more than onestorage element.

Finally, the actual storage reading and writing (and, potentially,additional mapping onto physical storage devices) occurs within thephysical storage system level 16, as illustrated in FIG. 1. Thus, forexample, the logical volume manager may map the file system level viewof data (as logical volumes or some other logical entity) into volumesizes corresponding to fixed physical storage segment sizes for storageon a physical device (e.g, block sizes). The physical storage systemlevel may then map the logical volume manager level (“logical”) volumesonto physical storage segments (e.g., hyper-volumes discussed below).

Logical volume managers have been implemented for use with the HP-UX byHP and by VERITAS for Solaris operating systems (i.e., V×VM), asexamples. The Symmetrix line of storage systems, available from EMCCorporation, of Hopkinton, Mass., is one system capable of mappinghyper-volumes onto physical devices. (The Symmetrix product line ofintegrated cached disk arrays is described in numerous publications fromEMC Corporation, including the Symmetrix model 55xx product manual,p-n200-810-550, rev.f, February, 1996.)

In the VERITAS Volume Manager, sold by VERITAS of Mountain View, Calif.,logical volumes are assigned tags, which consist of an identifier forthe host computer on which the volume manager is located and a timestamp indicating the time at which a logical volume was created. If alogical volume is electronically moved among the storage elements, it isassigned a new tag. Although logical volumes handled by this VolumeManager are not intended to be accessed by more than one host computerat any point in time, this has been done by manipulating the VolumeManager to provide tags for shared logical volumes to other hostcomputers.

In any event, in the above examples, the mapping of application leveldata into actual physical storage occurs across four levels: applicationlevel to file system level; file system level to LVM level; LVM level tophysical storage system level; and physical storage system level to theactual physical storage devices. More or fewer levels of mapping can bedone. In some systems, for example, only one level of mapping isperformed, e.g., mapping from the application level directly onto actualphysical storage devices. In many systems, the mapping stage at the LVMlevel is omitted. Similarly, in many systems, no mapping is done at thephysical storage level (e.g., data is stored directly onto actualdevices corresponding to the format of the preceding level and withoutany further mapping onto physical storage components.) As anotherexample, the file system level could be omitted and the applicationdirectly with logical (or actual) volumes.

FIG. 2A illustrates an example of the mapping that may be performed bythe logical volume manager 14 and the physical storage system 16, tostore data onto actual physical devices. The application/file system'sview of the storage system contemplates three separate storagedevices—volume A 20, volume B 21, and volume C 22. Thus, as far as thefile system level 12 can discern, the system consists of three separatestorage devices 20-22. Each separate storage device may be referred toas a “virtual volume,” “logical volume” or “virtual disk.” This reflectsthat the operating system's view of the storage device structure may notcorrespond to the actual physical storage system implementing thestructure (hence, “virtual”). Unlike the application level 10, however,the file system 12 perspective is as if the file system 12 were dealingwith raw physical devices or volumes.

As far as the file system level is concerned, the virtual volumes may bedivided up into “partitions,” which are continuous segments of storage.These partitions are, in fact, “virtual” partitions, because thepartition may actually be stored across a variety of physical storagesegments (e.g., hyper-volumes).

In FIG. 2A, the data is physically stored on the physical storagedevices 24-26. In this particular example, although there are threephysical devices 24-26 and three volumes 20-22, there is not a one toone mapping of the virtual volumes to physical devices. In thisparticular example, the data in volume A 20 is actually stored onphysical devices 24-26, as indicated at 20 a, 20 b and 20 c. In thisexample, volume B is stored entirely on physical device 24, as indicatedat 22 a, 22 b. Finally, volume C is stored on physical device 24 andphysical device 26 as indicated at 21 a, 21 b.

In this particular example, the boxes 20 a-20 c, 21 a-21 b and 22 a-22 brepresent contiguous segments of storage within the respective physicaldevices 24-26. These contiguous segments of storage may, but need not,be of the same size. The segments of storage may be referred to as“hyper-volumes,” and correspond to segments of physical storage that canbe used as components when constructing a virtual volume for use by thefile system. A hypervolume may be comprised of a number of “datablocks.” A data block is a unit of storage (e.g., a 512 byte block) thatis written or read at one time from the physical storage device.

Array management software running on a general purpose processor (orsome other mechanism such as a custom hardware circuit) 23 translatesrequests from a host computer (not shown) (made assuming the logicalvolume structure 20-22) into requests that correspond to the way inwhich the data is actually stored on the physical devices 24-26. Inpractice, the array management software 23 may be implemented as a partof a unitary storage system that includes the physical devices 24-26,may be implemented on a host computer, or may be done in some othermanner.

In FIG. 2A the array management software 23 performs the functions ofboth the logical volume manager 14 (if present) and the physical storagelevel 16, by mapping the file system's virtual volumes 20-22 intosegments that can be stored onto physical devices 24-26. The arraymanagement software 23 also performs the functions of the physicalstorage system level 16, by determining where to store the hyper-volumes20 a-20 c, 21 a-21 b and 22 a-22 b.

The physical storage devices shown in the example of FIG. 2A are diskdrives. A disk drive may include one or more disks of a recording media(such as a magnetic recording medium or an optical recording medium).Information can be written and read from this storage medium for storagepurposes. The recording medium is typically in the form of a disk thatrotates. The disk generally includes a number of tracks on which theinformation is recorded and from which the information is read. Eachtrack may include more than one “data block.” A data block is a unit ofdata that can be read as a single unit. A data block may be a 512 byteblock of data, an 8 k segment on a 32 k track, or some other structure.In these examples, the size of the block is fixed. In other cases, theblock may be of variable size, such as a CKD record. In a disk drivethat includes multiple disks, the disks by convention are described asthough they are stacked so that corresponding tracks of each diskoverlie each other. In this case, specification of a single track onwhich information is stored within the disk drive includes not onlyspecification of an individual track on a disk, but also which of themultiple disks the information is stored on.

To identify an individual data block, an address may include aspecification of the disk (which may consist of several “platters”) aspecification of the track within the disk (or “cylinder”), aspecification of the head (or which of the platters comprising the“disk”) and a specification of the particular data block within thetrack. The specification of the position of the data block within thetrack may, for example, be addressed as an offset, e.g., this is thethird data block appearing on the track. Thus, an address ofddcccch:offset may specify a block—disk dd, cylinder cccc, head h andthe specified offset. The physical storage devices for use with thepresent invention may, however, be formed in any other geometry,addressed in any other manner or even constitute a different type ofstorage mechanism.

FIG. 2B illustrates one example of mapping between the top level ofabstraction the application level—to the actual physical storage level.An application level file 200 includes visual information. Thisinformation is in the form of a conventional file and includes a seriesof bits.

When the application level file is mapped onto physical storage, theapplication level file may be converted into segments of the individualbits, e.g., segment 203. Thus, a segment of the application level file203 is mapped (for example according to the general mapping structuredescribed above with reference to FIG. 1) onto actual physical storagedevices 204-206. In this example, the first segment of bits in 203 inthe application level file 200 is mapped onto physical storage device204, at a portion 208 of the physical storage device 204. As shown inFIG. 2B, the individual segments of bits in the application level file200 may be mapped anywhere among a plurality of actual physical storagedevices. The granularity of the segments of bits (e.g., segment 203) maycorrespond to one of a variety of different levels. For example, thegranularity of the segments may be a 512 byte data block. In anotherembodiment, the granularity may correspond to the amount of data storedin a track of the physical storage devices 204-206 (when the physicalstorage devices are disk drives).

FIG. 2C illustrates an example of a logical object 27 that includes sixdata blocks or logical block elements 27 a-27 f. The logical objectitself may be any data structure or collection of data. For example, thelogical object could be a database table, a portion of a file systemfile, or a complete file system file, a logical volume or any otheridentifiable logical object. Each of the data blocks 27 a-27 f may be afixed size data block, or a varying size data block such as a CKDrecord.

In the example of FIG. 2C, the logical object is stored on a physicalstorage device 28. In this example, the storage device includes a numberof columns, each representing a track of a disk.

Each row of the physical storage device represents a physical data orblock element within the applicable column/track. For example, row 28 a,column 28 b, stores a data block corresponding to the logical blockelement 27 b. Track 28 b would store physical data blocks that have thecontents of logical block elements 27 a and 27 b. As can be seen fromFIG. 2C, the logical block elements can be stored in any order on thephysical devices.

While the physical storage device 28 is illustrated as a contiguousarray, this need not be the case. For example, each of the tracks, suchas column 28 b, may be stored on a different disk drive or be part of adifferent hypervolume.

In a system including an array of physical disk devices, such as diskdevices 24-26 of FIG. 2A, each device typically performs error detectionand/or correction for Is the data stored on the particular physicaldevice. Accordingly, each individual physical disk device detects whenit does not have valid data to provide and, where possible, corrects theerrors. Even where error correction is permitted for data stored on thephysical device, however, a catastrophic failure of the device wouldresult in the irrecoverable loss of data.

Accordingly, storage systems have been designed which include redundantstorage capacity. A variety of ways of storing data onto the disks in amanner that would permit recovery have developed. A number of suchmethods are generally described in the RAIDbook, A Source Book For DiskArray Technology, published by the RAID Advisory Board, St. Peter, Minn.(5th Ed., February, 1996). These systems include “RAID” storage systems.RAID stands for Redundant Array of Independent Disks.

FIG. 3A illustrates one technique for storing redundant information in aRAID system. Under this technique, a plurality of physical devices 31-33include identical copies of the data. Thus, the data M1 can be“mirrored” onto a portion 31 a of physical device 31, a portion 32 a ofphysical device 32 and a portion 33 a of physical device 33. In thiscase, the aggregate portions of the physical disks that store theduplicated data 31 a, 32 a and 33 a may be referred to as a “mirrorgroup.” The number of places in which the data M1 is mirrored isgenerally selected depending on the desired level of security againstirrecoverable loss of data.

In a mirror group, the copies are “linked.” That is, any update to onemirror causes an update to each other mirror in the group.

FIG. 3A shows three physical devices 31-33 which appear to be located inclose proximity, for example within a single storage system unit. Forvery sensitive data, however, one or more of the physical devices thathold the mirrored data may be located at a remote facility.

“RAID 1” is an example of data redundancy through mirroring of data. Ina RAID 1 architecture, a number of different mechanisms may be used fordetermining how to access and update data to improve, for example,performance of the storage system. In any event, a RAID 1 architecturecertainly has the ability to recover lost data. Unfortunately, the RAID1 architecture multiplies the cost of physical storage by the number of“mirrors” included in the mirror group.

FIG. 3B illustrates a solution that requires less added storage. In FIG.3B, data is stored at locations 34 a-34 d. In this particular example,the physical device 33 includes parity information P1 at 35 a, 35 b. Theparity information is generated by a simple exclusive-OR (“XOR”) of thecorresponding bits of data. Thus, the parity information P1 would begenerated by XORing the corresponding bits of the data D1 and data D2.

A variety of mechanisms are known for distributing the parityinformation on the physical devices. In the example shown in FIG. 3B,all of the parity information is stored on a single physical device 33.In other cases, the parity information may be distributed across thephysical devices.

FIG. 4 illustrates the concept that, within a given disk array, there isno need for all of the data to follow the same redundancy rule. In FIG.4, a first group of storage segments on physical devices 40-42 form amirror group 44. In the mirror group 44, the entire contents of a singlelogical volume (HV-A) are mirrored on three different physical devices40-42.

In FIG. 4, a single virtual volume is stored on the fourth physicaldevice 43, without any redundancy information, as indicated at 46.

Finally, a last group of data segments 45, on all four physical devices40-43, implement a parity redundancy scheme. In this particular example,the parity information is stored in segments of memory on two differentphysical devices 42-43, as indicated at 47 a and 47 b.

The storage system of FIG. 4 contains redundant information that permitsrecovery from errors, including use of a mirror for data located at aremote facility, that also permits recoveries from catastrophic failure.

FIG. 5 illustrates one system for additional backup, which may be usedor adapted in accordance with certain aspects of the present invention.In FIG. 5, a computer or client 50 performs its operations using storagesystem 52. The client 50 may be any conventional computing system, suchas a network client available from Sun Microsystems, and running theSolaris operating system (a version of Unix), an HP client running HP-UX(a Hewlett-Packard client, running a Hewlett-Packard version of the Unixoperating system) or an IBM client running the AIX operating system (anIBM version of Unix) or any other system with an associated operatingsystem. The storage system 52 may be any conventional storage system,including a Symmetrix storage system, described above. The client 50 maybe connected to many other devices over a network 56.

A backup storage system 54 is also attached to the network 56. Thebackup storage system 54 includes a backup storage device (which may bedisk drives, tape storage or any other storage mechanism), together witha system for placing data into the storage and recovering the data fromthat storage.

To perform a backup, the client 50 copies data from the storage system52 across the network 56 to the backup storage system 54. This processcan be explained in greater detail with reference to FIG. 1. The storagesystem 52 may correspond to the actual physical storage 16 of FIG. 1.For the client 50 to write the backup data over the network 56 to thebackup storage system 54, the client 50 first converts the backup datainto file data—i.e. gets the data from the physical storage system level16, and converts the data into application level format (e.g. a file)through the logical volume manager level 14, the file system level 12and the application level 10. Thus, an actual data file may becommunicated over the network 56 to the backup storage device 54. Whenthe backup storage device 54 receives the data file, the backup storagesystem 54 can take the application level 10 data file, convert it to itsappropriate file system level 12 format for the backup storage system,which can then be converted through a logical volume manager 14 leveland into physical storage 16.

This form of backing up data may be referred to as “logical—logical”backup. That is, the logical data is backed up on the backup storagedevice 54. The data to be backed up is presented independent of themanner in which it is physically stored on storage system 52 at thephysical storage system level 16, independent of the file system levelmechanisms on the client 50, and independent of how data is stored onthe backup storage device 54.

The EDM (EMC Data Manager) line of products is capable oflogical-logical backup over a network, as described in numerouspublications available from EMC, including the EDM User Guide (Network)“Basic EDM Manual”.

FIG. 6 illustrates one embodiment of an alternative structure for backupof data which may also be used in accordance with the present invention.In the embodiment of FIG. 6, a direct connection 60 is establishedbetween the storage system 52 and the backup storage system 54. In thisembodiment, the backup storage system may be a system as generallydescribed in EMC Data Manager: Symmetrix Connect User Guide, P/N200-113-591, Rev. C, December 1997, available from EMC Corporation ofHopkinton, Mass. The direct connection 60 may be a high speed datachannel, is such as a SCSI cable or one or more fibre-channel cables. Inthis system, a user may be permitted to backup data over the network 56,or the direct connection 60.

Whether the restore and backup process is done at a logical level or ata physical level, backups in the prior art require copying a completefile (or in some instances even more, such as an entire partition) forthe backup. Methods of backing up and restoring data on the system ofFIG. 6 are described in co-pending and commonly owned U.S. patentapplication Ser. No. 09/052,579, entitled “Logical Restore From APhysical Backup In A Computer Storage System,” filed Mar. 31, 1998, andnaming John Deshayes and Madhav Mutalik as inventors, and which ishereby incorporated herein by reference in its entirety.

FIG. 7 shows a storage system 70 that may be used as the storage system52 of FIG. 6. The client 50 may be connected to the storage device usinga channel or bus 71. The channel for communication with the client 50can be any suitable connection such as fibre channel, Small ComputerSystem Interface (“SCSI”) or Enterprise Systems Connection Architecture(“ESCON”). While only one communication channel 71 into the storagesystem 70 is shown in FIG. 7, other channels may be included. (While themethod and apparatus of the present invention may be described withreference to the storage system of FIG. 6 and the physical storagesystem (and associated features and methods) of FIG. 7, this is notintended to be limiting. The present invention has broader application.Certain aspects of the invention may be applied to any storage system.)Within the storage system 70 is a host adapter 72. In this particularembodiment, the host adapter 72 is responsible for managing andtranslating read and write requests from the host computer (e.g., client52 or backup storage system 54), which are based on the virtual diskstructure (e.g., from the file system or logical volume manager level),into one or more requests corresponding to how data is stored on theactual physical storage devices 76 a-76 d of the storage system 70.Thus, in this embodiment, the host adapter 72 implements at least someof the array management software 23 functions of FIG. 2. The hostadapter 72 can be implemented in any of a number of ways, includingusing a general purpose processor or a custom hardware implementation.In addition, multiple host adapters may be included to facilitate havingadditional I/O channels for the storage system 70.

The host adapter 72 communicates with the other components of thestorage system 70 using an interconnect such as bus 73. The bus 73 maybe any suitable communication element, including use of SCSI, ESCON, andother bus protocols.

Access to the physical storage devices 76 a-76 d is controlled throughthe use of disk adapters 75 a-75 d. The disk adapter 75 a-75 d can alsobe implemented using a general purpose processor or custom hardwaredesign. In the embodiment illustrated in FIG. 7, a disk adapter isprovided for each physical storage device. A disk adapter can, ofcourse, have more than one storage device attached to it. In addition,disk adapters may include secondary connections to the physical storagedevices of another disk adapter. This permits recovery from failure ofone disk adapter by shifting its functions to the second disk adapter.

In the embodiment of FIG. 7, reading and writing to the physical storagedevice 76 a-76 d through the disk adapters 75 a-75 d is facilitatedthrough use of a cache 74. The cache 74 may be a random access memoryhaving greater speed than the disk drives. When reading data, if thedata is being temporarily stored in the cache, the read request can befulfilled more quickly by taking the data from the cache 74. Similarly,when writing data, the data to be written can be stored in the cache.The other components of the system can proceed, while the data iswritten from the cache to the applicable physical storage device.

Any of a variety of mechanisms can be used to implement and manage thecache. An example of such a mechanism is included in U.S. Pat. No,5,537,568, entitled “System for dynamically controlling cache managermaintaining cache index and controlling sequential data access,” issuedon Jul. 16, 1996. Similarly, writes may be accomplished through thecache using any of a variety of mechanisms and strategies. One mechanismfor writing from the cache is to store the data to be written in thecache, and mark a “write pending” bit. When the write pending bit isencountered, the applicable data can be written to the disk. Thistechnique is described generally in U.S. Pat. No. 5,341,493, entitled“Disk storage system with write preservation during power failure,”issued on Aug. 23, 1994.

The cache may be divided into more than one area. For example, the cachemay include an area 74 a for storing data being read or written fromphysical storage devices 76 a-76 d. The cache may further include a“mailbox” area 74 b. The mailbox area 74 b may be used to facilitatecommunications among the disk adapters 75 a-75 d and with the hostadapter 72. For example, each disk adapter may have its own area withinthe mailbox 74 b. Each of the disk adapters 75 a-75 d can post or readinformation from the applicable mailbox area 74 b, to communicate statusand other information.

A remote adapter 78 may also be attached to the bus 73 of the storagesystem 70. The remote adapter may be employed for communication withremote data facilities (“RDF”), for example, connection to anotherstorage device to maintain a mirror redundancy group. One form of RDFlink and method of implementation is described in various publicationsavailable from EMC Corporation, including SYMMETRIX Remote Data FacilityProduct Manual, P/N 200-999-554, rev. B, June 1995. RDF embodiments arealso described in U.S. Pat. No. 5,544,347 (Yanai) which is herebyincorporated herein by reference in its entirety. It should beappreciated, however, that the present invention is not limited to theuse of RDF or to a system that employs SYMMETRIX disk arrays, and can beemployed with any of numerous other types of storage systems.

A service processor 77 may be coupled to the bus 73 of the storagesystem 70. The service processor 77 may include a display, keyboard andother I/O devices to permit an operator to use the service processor 77for configuring the components of the storage system 70 and for runningor initiating diagnosis and maintenance facilities.

While the method and apparatus of the present invention may be describedwith reference to the systems and concepts described above and in thediscussion of the related art, this is not intended to be limiting. Thepresent invention has broader application. Certain aspects of theinvention may be applied to any storage system. Accordingly, theinvention is only limited by the claims set forth below.

SUMMARY OF THE INVENTION

According to one embodiment of the present invention, a method ofspecifying a logical volume in the computer system is disclosed. Thecomputer system may have a plurality of storage elements and a pluralityof host computers. According to this embodiment, the method comprisessteps of assigning a logical volume identifier to the logical volume andassuring that the logical volume identifier is unique to the logicalvolume. According to this (and certain other) embodiments, the logicalvolume may be a conventional logical volume, a partition, ahyper-volume, a striped volume or a component of a conventional logicalvolume. As defined more fully below, a “logical volume” is an entitythat a host or application level computer use as a unit of storage.According to this embodiment, the same logical volume identifier may bemaintained when a logical volume is moved among storage elements in thecomputer system. The logical volume identifier may be assignedindependent of the location of storage for the logical volume.Descriptive information may be maintained with respect to logicalvolumes and their associated logical volume identifier, such asidentifying the entity that created the logical volume, identifyingentities that may access the logical volume, identifying access historyfor the logical volume and other information. The format of the logicalvolume identifier may correspond to the format for memory accessrequests made by host computers in the system. For example, the formatmay be that of a world wide name or a world wide name with a logicalunit identifier or may correspond to these formats by including these asa subfield. The logical volume identifier may be assigned independent ofthe host computer that created the logical volume.

According to another embodiment of the present invention, a method ofstoring and maintaining a logical volume is disclosed. According to thisembodiment, the logical volume is identified and then an ELVID (asdefined below) is assigned to the logical volume. The same ELVID ismaintained when the logical volume is electronically moved. A databaseof ELVIDs may be maintained at a storage management console orelsewhere.

According to another embodiment of the present invention, a computersystem is disclosed. According to this embodiment, a plurality ofstorage elements are provided together with means for assigning alogical volume identifier to a logical volume and means for assuringthat the logical volume identifier is unique to the logical volume towhich it is assigned.

According to another embodiment of the present invention, a computerstorage system is disclosed. According to this embodiment, at least onestorage element is provided, together with means for assigning ELVIDs tological volumes stored on the storage element. Such means may beprovided at a storage management controller, on one or more of thestorage elements or elsewhere.

According to another embodiment of the present invention, a storagemanagement controller is disclosed. According to this embodiment, anELVID assignor is provided to assign ELVIDs to logical volumes.According to this embodiment, an interface is also provided forcommunication with storage elements. The storage management controllermay further include an ELVID database manager or a logical volumemovement manager.

According to another embodiment of the present invention, a storagemanagement controller is provided. According to this embodiment, thestorage management controller includes an ELVID database manager and amemory to store an ELVID database.

According to another embodiment of the present invention, a method ofinventorying logical volumes in a computer storage system is disclosed.According to this embodiment, identifying information for each user of arespective logical volume is maintained for each of a plurality oflogical volumes in the computer storage system. “Users” may be anydefinable entity permitted to access memory, such as a host computer, anaccount on one or more host computers (e.g., a person logged into acomputer on their account) or an application program. For each of theplurality of logical volumes, a verification step is performed to assurethat logical volume is still in use. For example, if a logical volumehas not been accessed for a great period of time, the logical volume maynot need to be maintained on a primary storage element and can be movedto secondary storage. The step of verifying that the logical volume isstill in use can be performed as a continuous discrete process for thecomputer storage system, for logical volumes on a specific storageelement or in some other fashion.

According to another embodiment of the present invention, a storageelement is disclosed. According to this embodiment, the storage elementincludes a storage medium, an access manager module to maintainidentifying information for each user of logical volumes and a verifiermodule to perform verification. A verification module may include a timetracker to identify when a logical volume has not been accessed for anidentified period of time.

According to another embodiment of the present invention, a storagemanagement controller is disclosed. According to this embodiment, anaccess manager module and a verifier module are included.

According to another embodiment of the present invention, a method ofdetecting when one of a plurality of logical volumes no longer needs tobe stored on a respective storage element. According to this embodiment,accesses to the logical volumes are tracked, a logical volume that hasnot been accessed for a predetermined period of time is automaticallyidentified and, for such an identified logic volume, it is determinedwhether the identified logical volume still requires storage on itspresent storage element. This step of determining may comprise a step ofpolling users which have accessed the identified logical volume in thepast.

According to another embodiment of the present invention, a method ofidentifying a logical volume is provided. According to this embodiment,the logical volume is identified by reading an ELVID assigned to thelogical volume and using the ELVID to access identifying informationabout the logical volume. According to this embodiment, a database maybe maintained, for example, at a storage management controller. Theidentifying information may identify host computers that created thelogical volume or host computers authorized access the logical volume orhost computers that have accessed the logical volume (or which currentlyhave an open session which permits access of the logical volume).

According to another embodiment of the present invention, a computersystem is disclosed. According to this embodiment, a plurality ofstorage elements are provided together with means for reading a logicalvolume identifier and means for using the logical volume identifier toaccess identifying information about the logical volume.

According to another embodiment of the present invention, a logicalvolume identifier database is disclosed. According to this embodiment, acomputer readable medium is provided which storage a plurality oflogical volume identifiers, each identifier being unique to a respectivelogical volume stored among a plurality of storage elements. As above,additional identifying information may be included in the database.

According to another embodiment of the present invention, a method ofproviding a user with access to a logical volume is disclosed. Accordingto this embodiment, an ELVID for the logical volume is determined andprovided to the user. The user may be a host computer or an individualaccount on a host computer. The term “host computer” as used in thisspecification and claims includes both host computer sites, useraccounts on host computer sites and other entities that are permitteduser level access to data stored on storage elements in the computersystem. According to this embodiment, the method of providing access caninclude a step of maintaining a database of ELVIDs stored on a pluralityof storage elements. The maintaining step may be performed on a hostcomputer or a storage management controller.

According to another embodiment of the present invention, a method ofaccessing logical volumes is disclosed. According to this embodiment, anELVID is determined and used to access the logical volume.

According to another embodiment of the present invention, a hostcomputer is disclosed. According to this embodiment, the host computerincludes a processing unit (any hardware unit which permits processing,such as a microcontroller, microprocessor, or special hardware designedfor a particular host computer) and an ELVID interface module totranslate requests for accesses to a logical volume to an ELVID for thelogical volume.

According to another embodiment of the present invention, a hostcomputer is disclosed. According to this embodiment, the host computerincludes a processing unit and an ELVID module to translate an ELVID fora logical volume to a physical storage location for that logical volume.

According to another embodiment of the present invention, a storagemanagement controller is disclosed. According to this embodiment, thestorage management controller includes an access management module toprovide access to logical elements by providing a physical storageaddress associated with an ELVID for the respective logical volume.

According to another embodiment of the present invention, a computersystem is disclosed. According to this embodiment, the computer systemincludes a plurality of host computers, a plurality of storage elementsand means for assigning ELVIDs to logical volumes.

According to another embodiment of the present invention, a method ofmoving a logical entity from a first storage element to a second storageelement is disclosed. According to this embodiment, a copy of thelogical entity is created on the second storage element (for example, bymirroring). All reads for the logical entity are moved to the locationof the second storage element. After all of the reads have been movedwrites are moved to the location for the logical entity on the secondstorage element. If a mirror is used, the mirror may be maintained whileall of the reads are moved. The logical entity may be a logical volume,such as a conventional logical volume, a component of a conventionallogical volume, a hyper volume, a partition or a striped volume. Themethod according to this embodiment may include a step of determining anELVID for the logical entity.

According to another embodiment of the present invention, a hostcomputer is disclosed. According to this embodiment, the host computerincludes a processing unit and a memory interface module. According tothis embodiment, the memory interface module permits access to a logicalentity to be made to one physical storage location for read requestwhile write requests are made to a different physical storage location.According to some embodiments, the memory interface module can includean ELVID interface module to translate requests for access to a logicalvolume to an ELVID for that logical volume. According to otherembodiments, the memory interface module may instead or in additioninclude an ELVID interface module to translate an ELVID to a physicalstorage location for a logical volume.

According to another embodiment of the present invention, a storagemanagement controller is disclosed. According to this embodiment, thestorage management controller includes an interface module tocommunicate with storage elements and an entity movement manager tocontrol separate moving of read locations and write locations for aspecified logical entity.

According to another embodiment of the present invention, a computersystem is disclosed. According to this embodiment, the computer systemincludes a plurality of host computers, a plurality of storage elementsand means for separately moving reads for a logical entity and writesfor a logical entity.

According to another embodiment of the present invention, a method ofmoving a logical volume from a first storage element to a second storageelement is disclosed. According to this embodiment, an ELVID is providedfor the logical volume. A copy of the logical volume is provided on thesecond storage element. The physical storage location on the secondstorage element is associated with the ELVID to cause accesses to thelogical volume to be made to the second storage element. The step ofassociating may be performed, for example, by the host computer, astorage management controller or both (for example, with the storagemanagement controller initiating the change in the host computerperforming the change).

According to another embodiment of the present invention, a method ofrolling back a logical volume to a previous state is disclosed.According to this embodiment, a copy of the logical volume in a previousstate is provided. An ELVID for the logical volume is provided. The rollback is completed by associating a physical storage location for thecopy with the ELVID, to cause accesses to the logical volume to be madeto the physical storage location for the copy.

According to another embodiment of the present invention, a method ofproviding for and implementing immediate restoration of the logicalvolume is disclosed. According to this embodiment, a restoration copy ofthe logical volume is maintained at a physical storage location on astorage device. An ELVID is provided for the logical volume. Therestoration is completed by changing a physical storage locationassociated with the ELVID for the logical volume to correspond to thephysical location on the storage device for the restoration.

According to another embodiment of the present invention, a storagemanagement controller for a storage system with a plurality of storageelements is disclosed. According to this embodiment, the storagemanagement controller includes an interface module to communicate withthe storage elements and a logical volume movement module to move thelocation of logical volume access request by updating entries in atleast one ELVID database.

According to another embodiment of the present invention, a computersystem is disclosed. According to this embodiment, at least one hostcomputer and a plurality of storage devices are included in the computersystem. The computer system further includes means for changing thelocation of access requests, including means for changing a physicalstorage location address associated with an ELVID for the logicalvolume.

Accordingly to another embodiment of the present invention, a method ofaccessing a logical volume stored on at least one of the plurality ofstorage elements is disclosed. According to this embodiment, an ELVIDfor the logical volume is specified. A physical storage address for thelogical volume is also specified. This embodiment further includes astep of verifying that the ELVID corresponds to the physical storageaddress. The verification step may be performed, for example, by a hostcomputer, a storage management controller, a storage element or somecombination of these.

According to another embodiment of the present invention, a method ofaccessing a logical volume stored on at least one of a plurality ofstorage elements is disclosed. According to this embodiment an ELVID forthe logical volume is specified and a physical storage address for thelogical volume is specified. The method includes a step of using theELVID to assure that an entity requesting access to the logical volumeis authorized to do so.

According to another embodiment of the present invention, a hostcomputer is disclosed. According to this embodiment, the host computerincludes a processing unit and an ELVID interface module to transmitaccess requests for logical volumes, access requests including an ELVIDfor the logical volume and a respective physical storage location forthe logical volume.

According to another embodiment of the present invention, a storagedevice is disclosed. According to this embodiment, the storage deviceincludes a storage medium and an ELVID verifier module, to verify accessrequests.

According to another embodiment of the present invention, a storagedevice is disclosed. According to this embodiment, the storage deviceincludes a storage medium and an ELVID authorization module to verifyaccess requests.

According to another embodiment of the present invention, a computersystem is disclosed. According to this embodiment, the computer systemincludes at least one host computer, a plurality of storage elements,means for associating ELVIDs with requests for access to logical volumesand means for verifying that access requests to physical storagelocations are made for the appropriate logical volume identified by arespective ELVID.

According to another embodiment of the invention, a computer system isdisclosed. According to this embodiment, the computer system includes atleast one host computer, a plurality of storage elements and means forverifying that access request logical volumes having an associated ELVIDare made by an entity authorized access logical volume.

Each of the above disclosed invention embodiments may be used andapplied separately and independently, or may be applied in combination.Description of one aspect of the invention is not intended to belimiting with respect to other aspects of the invention. In addition,application of combinations of the above disclosed embodiments may alsobe inventive; systems including many of these combinations is describedmore fully below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of conversion of application level data tostorage in a physical system and vice versa.

FIG. 2A illustrates an example of the relationship between logicalvolumes and physical storage devices.

FIG. 2B illustrates an example of mapping a logical file onto a physicalstorage system.

FIG. 2C illustrates another example of mapping a logical object onto aphysical storage device.

FIG. 3A illustrates an example of mirroring on different physicalstorage devices.

FIG. 3B illustrates an example of redundant parity information onphysical storage devices.

FIG. 4 illustrates an example of multiple redundancy groups within asingle array of storage devices.

FIG. 5 illustrates an example of a backup storage system.

FIG. 6 illustrates one embodiment of a backup storage system thatincludes a mechanism for direct backup of data on the primary storagesystem.

FIG. 7 illustrates an example of a storage system.

FIG. 8 illustrates one example of a computer storage system structuredto have an enterprise host domain and an enterprise storage domain ornetwork, according to one embodiment of the present invention.

FIG. 9 illustrates another example of a computer system including anenterprise host domain and an enterprise storage domain, according toone embodiment of the present invention.

FIG. 10 illustrates one example of some of the components of a computersystem that includes a host domain and a storage domain, according toone embodiment of the present invention.

FIG. 11 illustrates another embodiment of components of a computersystem that is divided into a host domain and a storage domain,according to one embodiment of the present invention.

FIG. 12 illustrates one embodiment of a computer system that includes ahost domain, storage domain and storage management console.

FIG. 13 illustrates one embodiment of a method that assigns enterpriselogical volume identifiers (“ELVIDs”) as defined below.

FIG. 14 illustrates one embodiment of a computer storage systemcorresponding to FIG. 12 that includes ELVID configuration tables.

FIG. 15A illustrates one embodiment of the method of accessing logicalentities using an ELVID or ELVID and logical unit number.

FIG. 15B illustrates an example of application of the method of FIG. 15Ato access data over a computer network.

FIG. 16 illustrates one embodiment of a method for accessing data usingan ELVID and mapping the ELVID to a physical location.

FIG. 17 illustrates one embodiment of a method for verifying accessusing an ELVID.

FIG. 18 illustrates an example of applying the method in FIG. 17 to oneembodiment of a computer system that uses ELVIDs.

FIG. 19 illustrates one embodiment of a method for moving logicalentities in a system that uses ELVIDs.

FIG. 20 illustrates one embodiment of moving an ELVID to a new physicallocation.

FIGS. 21A-21C illustrate application of the method of FIG. 20 in thecontext of one embodiment of a computer system.

FIG. 22 illustrates one embodiment of a method for restoring a backupcopy of a logical entity using an ELVID.

FIG. 23 illustrates one embodiment of a computer system that uses ELVIDsfor performing various functions.

DETAILED DESCRIPTION

Storage as a Domain.

The architectures illustrated in FIG. 5 and FIG. 6 may be viewed asfocusing on a network model for storage, or a “network-centric” system.In such a system, the focus of data transfer is movement of logical dataacross a network. Moreover, the storage system 52 and backup storagesystem 54 are typically associated with a single client or host 50architecture.

An alternative model focuses on a separation of the client or hostdomain and the storage domain.

FIG. 8 illustrates one example of a system which segregates the hostdomain from the storage domain. In FIG. 8, a number of host computers 80are included in an enterprise host domain 80 a. “Host computers” can beany type of computers, operating systems and data managementapplications. For example, one host computer 80 may be a Hewlett Packard9000 computer system running an HP-UX Operating System. Another hostcomputer 80 can be a Sun Sparc Station running a Solaris operatingsystem. The combination of a host, operating system and applicable datamanagement application is referred to as a “platform.” Each of the hostcomputers 80 may constitute a different platform interfacing with thestorage network 89.

The host computers 80 in the enterprise host domain 88 may be connectedover a network. This network may include switching nodes 81, althoughany other form of network may be used.

In the embodiment of FIG. 8, the host computers 80 are coupled to theenterprise storage network or domain 89 through a network or directly toprimary storage nodes 82. A primary storage node is a memory device thatcan store a significant amount of data for use by the host 80. Forexample, a Symmetrix system, such as the one described above withrespect to FIG. 7, may be used as a primary storage node, although thisis not intended as limiting.

In the embodiment of FIG. 8, each host computer is coupled to a subsetof primary storage nodes 82, for use as a main memory for that hostcomputer. For example, host computer 80 a is coupled directly to primarystorage node 82 a. The host computer 80 a may rely on primary storagenode 82 a for most of its memory intensive functions, such as foraccessing a very large database.

The primary storage nodes 82 may also be coupled together through anetwork. In the example of FIG. 8, the network includes link 85 andswitch network 84. The switch network 84 may, for example, be a fibrechannel network. The link 85 may be an RDF link over an ESCON line orany other channel.

The network between primary storage nodes may serve two purposes. Thenetwork may permit transfer of data between primary storage nodes. Forexample, a database being manipulated by host 80 a, and stored inprimary storage node 82 a, may be transmitted to primary storage node 82b for use by host 80 b. By transmitting the database across theenterprise storage network (using link 85 or switching network 84), thecomputational resources of the host 80 a, 80 b, and the availablebandwidth in the enterprise host domain network, can be preserved.

The enterprise storage network 89 may also include a secondary storagenode 87. The secondary storage node may be used for backup functions,hierarchical storage management, virtual disks and other functions.Thus, the secondary storage node 87 may be coupled to a tape storageunit 83. The secondary storage node 87 would coordinate sophisticatedtransfer of data from the primary storage nodes 82 to the tapes storedin a tape storage unit 83. (Other embodiments may use additional oralternative media for secondary storage.)

FIG. 9 illustrates one embodiment of a computer network. In thisembodiment, an enterprise host domain 97 is provided. The enterprisehost domain 97 includes a variety of host computers 90 a-90 e. The hostcomputers may include different platforms and different correspondingmechanisms for accessing and storing data in the memory. For example,host computer 90 a is a Hewlett Packard HP 9000 computer. Host computer90 c is a Sun Sparc Station which may be running a Solaris OperatingSystem. The host computers may communicate with each other across anetwork 96. Such a network can be one of many computer networks knownand applied for connecting computers.

In the embodiment of FIG. 9, each host computer 90 a-90 e is connectedto a primary storage node 92 a-92 c. In this embodiment, each primarystorage node 92 a-92 c is an iterative cached disk array, such as aSymmetrix memory system such as the one described above with respect toFIG. 7, although this is not intended to be limiting. Thus, for example,host computer 90 a interfaces primarily with storage node 92 a.Similarly, host computer 90 b uses primary storage node 92 a as aprimary source of its data.

In the embodiment of FIG. 9, the host computer 90 a is connected to theprimary storage node 92 a over a high speed fibre channel 91 a. The host90 b, however, is connected to the primary storage node 92 a over astandard SCSI connection. Each of the hosts 90 a and 90 b are coupled tothe same primary storage node 92 a. Other mechanisms could be used toconnect the host computers 90 a-90 e to the primary storage nodes 92a-92 c. For example, a complete switched network could be employed, forany of the host computers to access any of the primary storage nodes 92a-92 c.

Each of the primary storage nodes 92 a-92 c may also be coupled togetherusing a network. In the example of FIG. 9, the only link among theprimary storage nodes is an ESCON remote data facility (ESCON “RDF”)link 93 g, although any other type of link may be used. Such a link maybe used for transferring of data or maintaining a mirror of data eitheron-line or as a periodically updated mirror. Such a link may beimplemented as described in U.S. Pat. No. 5,544,347 (Yanai), which isincorporated herein by reference in its entirety. Each of the primarystorage nodes 92 a-92 c may be coupled together using any othermechanism. For example, an RDF link could be used to fully connect eachof the primary storage nodes 92 a-92 c. In the alternative, a switchnetwork could be used, assuming that the network is of sufficiently highspeed to support the data operations among the primary storage nodes 92a-92 c.

The storage network 98 in the embodiment of FIG. 9 further includes asecondary storage node 94. The secondary storage node is used for backup(and other) functions, for example by storing and restoring informationto and from a tape library 95. In the embodiment of FIG. 9, each of theprimary storage nodes is connected or connectable (by a network) to thesecondary storage node 94. In this example, primary storage nodes 92 band 92 c are coupled to secondary storage node 94 each using an RDF link(93 c and 93 d respectively) which may be implemented as describedabove.

The primary storage node 92 a is connected (together with other primarystorage nodes, not shown) to the secondary storage node 94 over aswitched network, which will permit each of the systems to access thesecondary storage node 94.

Using an RDF (or other) link that permits high speed transfer of dataover long distances, the primary storage nodes 92 a-92 c and thesecondary storage device 94 may be physically located at great distancesapart.

Of course, other topologies and other mechanisms may be used withoutdeparting from the scope of the invention.

Many of the applications for computers now focus as much or more onmemory than on the ability of the system to perform computations. Forexample, access to very large databases has become an extremelyimportant and valuable application for computers.

In the past, the focus of computer systems has been on interconnectinghost computers each having their own associated memory, or providingnetwork access to a single memory. This focus demands host computer andnetwork resources.

In the storage-centric model, however, the storage component of thecomputer system is elevated to a status of equal importance. In such amodel, the storage components of the system are capable interacting witheach other with less involvement from the host domain. For example, itmay be desirable to permit mirroring across one or more primary storagenodes. Similarly, data or logical objects may need to be copied from oneprimary storage node to another primary storage node. Where additionallevels of backup are desirable, the primary storage nodes may alsotransfer data to a secondary storage node for backup purposes. Theprimary storage nodes may, correspondingly receive data from thesecondary storage nodes for restore. In a storage centric model, some orall of the resource intensive functions in such a system can be movedout of the host domain. Certain embodiments following this model canpreserve host domain resources, increase scalability of memory (byadding to the storage domain without as much concern about affect onhost domain resources) and reduce dependence on the particular platformsof the hosts in the host domain.

Under a storage-centric model of computer storage, it may be desirableto permit as much of the data transfer process to be performed withinthe storage network—and without requiring resources from the hostdomain. Accordingly, the primary storage nodes and the secondary storagenodes in the network may include sufficient intelligence to handleaspects of the data transfer process. For example, the primary storagenodes may be capable of managing the transfer of identified physicalelements in a logical object even when those physical elements arestored in disparate locations within the primary storage device. In astorage centric model of a computer system, it may be desirable to movesome (or as much as possible, in some cases) of the data transferfunctions to be performed using resources among primary and secondarystorage nodes within the storage domain.

The computer system may include a storage management application(“SMAPP”) for managing manipulation of storage within the storagedomain. The SMAPP can be implemented using software on the hostcomputers, primary storage nodes, a separate storage controller or insome combination of these, as described below with reference to FIGS. 10and 11, below.

The storage management application can be implemented using threeprimary components—a management component, server component and clientcomponent.

In this example, the management component controls configuration of thebackup, control and monitoring of the backup and copying processes inthe storage domain. The management component also tracks location ofcopies of logical objects in the storage system including, for example,what tape or tapes contain backups of each particular logical object.

The server component of this example controls the hardware functions ofthe memory process, such as acts of mounting and dismounting tapes,opening and closing, reading and writing tapes and other memory media.

The client component of the SMAPP handles manipulation andidentification of the backup or copy-from source. For example, theclient component is responsible for identifying the applicable logicalobject (e.g., file system, file or database) and determining whatoperating system level (or logical volume manager level) physicalelements are involved. (As described above, an additional layer ofmapping may be performed within the storage domain at the primarystorage element of 111. For example, if the primary storage element 111is a Symmetrix product as described above, the identified physicaltracks may be re-mapped within the primary storage element 111.)

FIG. 10 illustrates one example of a portion of a computer system havinga host domain and a storage domain. In the example, only one host 110 isshown in the host domain. In addition, only three components are shownin the storage domain. These are the primary storage element 111 (whichmay be, for example, a Symmetrix disk array), a secondary storageelement 112 and a tape library unit 113. As described above, additionalstorage elements may be included, coupled together by a network. Forsimplicity, the example of FIG. 10 shows only one element from each ofthree different storage levels—host, primary storage element andsecondary storage element.

In the example of FIG. 10, a storage management application (“SMAPP”)114 is primarily resident on the host computer 1 10. Thus, the hostcomputer would include an Application Programming Interface (“API”)which would permit management of copying, backup and restore (and other)operations. In addition, the storage management application 114 on thehost 110 includes a server component 11 5 b. Again, the host wouldinclude an API permitting management of server operations. Finally, thestorage management application 114, in this example, includes a clientcomponent 11 5 c. The client component would be responsible foridentifying and manipulating logical objects and identifying (from theoperating system or logical volume management level view of) thephysical elements that comprise the logical object.

For simplicity, the operation of performing a backup from the primarystorage element 111 to the secondary storage element 112 will bedescribed. A similar process would apply for setting up mirroring orcopying functions between primary storage elements in a network.

In this example, the primary storage element includes an SMAPP interface116 a. Similarly, the secondary storage element 112 includes an SMAPPinterface 116 b. The copying of a logical object from the primarystorage element 111 to the secondary storage element 112 in theembodiment shown in FIG. 10 may proceed as follows. First, a “virtualcircuit” or “connection” is set up between the primary storage element111 and the secondary storage element 112. This may be a virtual circuitestablished through a network coupling the primary storage element tothe secondary storage element 112 (including a single RDF link betweenthe primary storage element 111 and the secondary storage 112, forexample). In addition to establishing a physical connection between thenodes, the virtual circuit identifies a session for copying a series ofdata (comprising, e.g., the logical object) over the identifiedconnection.

Thus, the management component 115 a on the SMAPP 114 on the hostcomputer 110 may begin a backup session by instructing the primarystorage element to establish a virtual circuit with the secondarystorage element 112. The actual establishment of the virtual circuit maythen be performed by the SMAPP interface 116 a of the primary storageelement 111 in combination with the SMAPP interface 116 b of thesecondary storage element 112.

The client component 115 c of the host computer 110 identifies a logicalobject for backup. The client component 115 c then maps that logicalobject to the operating system (or a logical volume manager level) setof physical elements. This mapping may be performed in one step. Theclient component 115 c of the host 110 may then identify the elementsfor copying to the primary storage element 111, as communicated throughthe SMAPP interface 116 a.

The server component 115 b of the host 110 would identify and mount theappropriate tapes in the tape library unit 113. In this particularexample, the server component 115 b performs these commands by passingthem to the SMAPP interface 116 b of the secondary storage element 112,through the SMAPP interface 116 a of the primary storage element 111,which then mounts the tapes.

The actual performance of the backup process may proceed, withoutfurther control by the host 110 of the host domain (except, in someembodiments, monitoring the process and managing the backup media, e.g.,controlling changing of tapes in a tape drive). The primary storageelement 111 may copy the identified physical segments to the secondarystorage element 112.

FIG. 11 illustrates an alternative structure for control of the storagedomain of a computer system according to the present invention. In thisexample, a storage network controller 118 a is connected to the host110, primary storage element 111 and secondary storage element 112through a network 119. This network, for example, may follow the TCP/IPprotocol. The storage network controller 118 a may be any hardware, orhardware and software, combination capable of performing the requisitefunctions. For example, the storage network controller 118 a may be acomputer running a windows NT operating system, with suitableapplication software for performing the SMAPP functions.

In this example, a significant portion of the SMAPP software is residenton the storage network controller 118 a. Thus, the SMAPP 118 b of thestorage network controller 118 a includes a management component and aserver component. Thus, management of the hardware and media can beperformed by the storage network controller 118 a, independent of thehost computer 110.

In this example, the host 110 includes an SMAPP 117 to perform clientfunctions. Thus, logical to physical mapping is still performed in thehost domain by the host computer 110. As the client component of theSMAPP 117 is responsible for identifying logical objects and performinglogical to physical mapping, this can be a sensible arrangement. Thelogical to physical mapping depends on the particular host platform andthe host necessarily has elements capable of performing the requisitemapping.

In other embodiments, however, the client component can be included inthe storage network controller 118 a, or in a separate device capable ofperforming logical to physical mapping for one or more platforms. Wherethis is done, the identification and transfer of data for copying andbackup purposes can be performed completely separately from the hostdomain. In many systems, however, it will be more efficient to use thememory mapping mechanisms (client component) on the host computer.

Other arrangements of the SMAPP software are possible. For example, thecomponents of the SMAPP software may be distributed across the primarystorage elements in the storage domain, the secondary storage element orelements in the host domain or some combination thereof.

FIG. 12 illustrates another simplified example of a computer systemdivided into a host domain 120 and storage domain 121. This simpleexample facilitates description of one embodiment of the presentinventions, but is not intended to be limiting.

In FIG. 12, the host domain 120 includes two host computers—122 a and122 b. These host computers may be any computer system for applicationprocessing, as described generally above.

The storage domain 121 of this particular example includes a network123, enterprise storage management controller 124 (which can correspondto the storage network controller described above) and two primarystorage elements 125 a and 125 b. The primary storage elements 125 a,125 b handle the major storage requirements for the host computers 122a, 122 b. In some systems, the primary storage elements may be generallyaffiliated with particular host computers. For example, host computer122 a may use primary storage element 125 a for most of its needs, anduse primary storage element 2 125 b only in special circumstances or foroverflow. Similarly, host computer 122 b may rely primarily on primarystorage element 125 b. In this circumstance, the network 123 whichcouples the host computers 122 a, 122 b to the primary storage elements125 a and 125 b may be configured to allow more rapid access betweenhost computers and affiliated storage elements. Indeed, the network 123may include one or more subnetworks, for example including a directconnection from host 122 a to the primary storage element 125 a, inaddition to other connections to one or more other networks.

In the example of FIG. 12, a logical volume A 126 is stored on a primarystorage element 1 125 a. Additional copies of logical volume A 126 maybe stored elsewhere in the storage domain 121, for example mirrored onprimary storage element 2 125 b. In addition, subcomponents (e.g., hypervolumes) could be stored across more than one storage element. In anyevent, for this example, the logical volume A 126 is stored entirely onprimary storage element 1 125 a.

Accordingly, whenever host 1 122 a or host 2 122 b wishes to access datain volume A 126, the access request will be transmitted over the networkto primary storage element 1 125 a.

In the event that the connection by which host 1 122 a accesses andreceives data from primary storage element 1 125 a is over a networkthat allows access to more than one device, some addressing scheme isapplied for the host 122 a to specifically address the primary storageelement 125 a. For example, if the network 123 includes a fibre channelnetwork that permits access to a primary storage element 125 a, theprimary storage element 125 a may be assigned a 64-bit worldwide name(“WWN”). The worldwide name is an address for the primary storageelement 125 a over the fibre channel network. Of course, otheraddressing schemes may apply for the applicable network, such as the useof an IP address.

Thus, when the host 122 a seeks to access primary storage element 125 a,the host will specify an address corresponding to the applicable primarystorage element, here an address corresponding to primary storageelement 125 a. In addition, the host 122 a will need to specify the datato be accessed. In most embodiments, this will include an identifier ofthe particular logical entity being accessed, such as an identifier of alogical volume to be accessed. In the example of FIG. 12, the host 122 amay specify that logical volume A 126 is to be accessed.

In some embodiments of the present invention, specification of theparticular logical entity may be through a logical unit numberidentified by the host computer 122 a. In this case, and continuing withthe example of a fibre channel network, accessing data from primarystorage element 125 a by host 122 a may use specification of a storageelement address (here, a worldwide name) and an applicable logicalentity (here, volume A, as identified by a logical unit number).

It may become necessary, for a variety of reasons, to move the locationof the primary copy of volume A 126. For example, if primary storageelement 1 125 a is to be replaced in the system, volume A 126 would haveto be moved to a replacement or other primary storage element. In othercircumstances, it may become desirable to move volume A 126 to adifferent primary storage element in order to accommodate load balancingeither among the primary storage elements 125 a, 125 b or in the network123. As one example, more than one host may need to share a volume.Depending on which hosts are accessing the volume (and thecharacteristics, such as frequency, of access), the best location forthe volume may vary. In other embodiments, it may be desirable to movesome portions of volume A 126 to another primary storage element 2.

This process can, however, be complicated. When host 1 122 a accessesthe volume A 126 stored in primary storage element 1125 a, host 1 issuesan access request that is formatted to be interpreted by the appropriateswitches (or other devices) in network 123 to route the access requestto the correct destination (primary storage element 1 125 a). If volumeA 126 is moved to a different primary storage element, e.g., primarystorage element 2 125 b, access requests from host 1 122 a would berouted to an incorrect destination. Accordingly, movement of volume A126 in primary storage element 125 a to primary storage element 125 bwould require substantial involvement and management by the hostcomputers 122 a, 122 b.

ELVIDs

According to certain embodiments, the above problem is addressed (inpart) through assignment of enterprise logical volume identifiers(“ELVIDs”). An ELVID is a unique identifier associated with a logicalentity, such as a logical volume, used by a host in the host domain 120.For example, logical volume A 126 in FIG. 12 may be assigned an ELVIDthat is a unique identifier for the logical volume stored on primarystorage element 125 a.

As described more fully below, in one embodiment, the ELVID may be usedto translate between the logical name of the entity (from theapplication view) and its physical location (from the storage domainview; although further translation into a physical location may beprovided within the storage domain, such as mapping to a location withina storage device). The translation from logical name to physicallocation may be implemented using software, firmware or hardware betweenthe logical volume manager (or file system, where there is no logicalvolume manager) level and the physical storage level (although thistranslation can also be implemented with software, firmware or hardwarelocated at a variety of different levels, or among levels, as well).

Although ELVIDs are described below with respect to identifying logicalvolumes at the logical volume manager and physical storage system level,ELVIDs may be assigned to other entities, such as the other logicalobjects described above, including components of logical entities suchas hypervolumes, also described above and also being “logical objects”themselves. Accordingly, an “ELVID” is simply a unique identifierassociated with a logical object, segment or grouping of data. The ELVIDidentifier should be unique for the storage domain in the applicablecomputer system (no matter how that storage domain is implemented,through use of one or more primary storage elements or otherwise). TheELVID may also be independent of the actual physical location at whichthe data is stored. This independence need not, however, be completeindependence. For example, the ELVID may be assigned based on theinitial location of the logical entity. If the logical entity were tomove, however, the ELVID would remain the same. Thus, ELVID is“independent” of physical location in the sense that, once assigned, theELVID would remain the same even if the physical location of the datachanges. In some embodiments, the ELVID may reflect the identity of astorage domain or subgroup of storage elements. As long as the ELVIDremains the same when the data is moved it is “independent” of physicallocations.

FIG. 13 illustrates one embodiment of a method for assigning ELVIDs in acomputer system. At step 130, the logical entity to be assigned an ELVIDis identified. For an existing system, this would correspond toidentifying a logical entity being used by a host application and forwhich memory is required (or has been allocated) in the storage domainof the computer system. In a running system, this may correspond to theopening of a new file system or other structure to be used by the hostcomputer. The logical entity could be any logical object, includinghyper volumes and other logical entities used in the memory hierarchy ofthe computer system. Since applications generate data for manipulation,the ELVIDs may be associated, for example, with entities at theapplication level, such as a particular database or file. Alternatively,or in addition, ELVIDs may be assigned to logical entities at the filesystem level. For example, ELVIDs may be assigned to logical volumes orpartitions within logical volumes. Similarly, ELVIDs may be assignedonly or in addition to logical entities at the logical volume managerlevel, such as hyper volumes. In many cases, the logical entity to beassigned an ELVID is identified at the host computer—the computersprimarily responsible for using, reading and writing the data in eachlogical entity. Because storage elements and the enterprise storagemanagement console or storage network controller may includeintelligence and may constitute complete computer systems themselves,these entities may also define or identify logical entitles to beassigned an ELVID.

At a step 131, the ELVID is assigned to the logical entity. As describedabove, the ELVID is preferably a unique identifier for the logicalentity. In the embodiment of FIG. 12, the enterprise storage managementconsole 124 (or other single agent within the system) may be used forassigning all ELVIDs. By using an enterprise storage management console,the uniqueness of the ELVIDs for the entire storage domain may be moreeasily assured. In other embodiments, however, the enterprise storagemanagement functions could be distributed across the host computers,primary storage elements or other devices in the complete computersystem.

According to one embodiment of the present invention, the number of bitsfor the ELVID is selected to be the same as the number of bits used bythe host computer for identifying storage elements. For example, if afibre channel is being used to access storage elements and a worldwidename of 64 bits is used to address the storage element, 64 bits may alsobe used to address the ELVID. In another embodiment, the ELVID mayinclude the same number of bits as the addressing scheme for thephysical storage elements (e.g., worldwide name) plus the number of bitsfor a logical unit number. By using the same number of bits as used bythe host computer for identifying storage elements or logical units onstorage elements, some host operating systems may be able to more easilyuse an ELVID in the ordinary identifier mechanism for accessing logicalentities such as logical volumes.

Some operating systems may employ a different mechanism that has fewerbits for identifying logical entities. For example, an operating systemmay allow only logical unit numbers to identify logical units. For thesesystems (as well as other systems), an ELVID having a greater number ofbits may still be assigned, with an additional mapping step between theELVID and the logical unit number, as described below.

At a step 132, the actual physical memory that will (at least initially)store the logical entity is identified and/or allocated. This step maybe performed in a conventional manner. In the embodiment of FIG. 12,however, the enterprise storage management console 124 may beresponsible for allocating memory. This allocation can be performedusing any number of optimization algorithms for determining the bestlocation for the logical entity. In other embodiments, another mechanismmay be used. For example, if a primary storage element is specificallyaffiliated with a host computer, physical memory may always be allocatedon that primary storage element, or at least given priority forassignment on that storage element.

At a step 133, the ELVID is mapped to the allocated memory. This step ofmapping configures the mechanism that permits a host making a requestfor access to a logical entity (e.g., by ELVID) to locate or verify thephysical location of that logical entity.

FIG. 14 illustrates one example of how an ELVID may be mapped tophysical memory in an example system. The example system correspondsroughly to the computer system described above with reference to FIG.12. In this example, the enterprise storage management console 124 hasassigned an ELVID, {VOL. A}, for a logical volume A to be accessed (atleast initially) by host 1 122 a.

In this example, the enterprise storage management console 124 retains adatabase tracking logical entities that have been assigned ELVIDs in thestorage domain 121. “Database” includes any structure for storing data,including tables, arrays, linked lists, objects and other constructs andmay be stored on a single, or across multiple, memories. In the exampleof FIG. 14, a table is shown but this is not intended as limiting. Thetable 141 includes a row 141 a that illustrates the entry for the ELVIDassigned to the logical entity volume A. The assigned ELVID isidentified in the first column 141.

In the example of FIG. 14, the table 141 includes a column that allowsdescription information for the applicable ELVID. For example, thedescription field may include a description of the applicable logicalentity (e.g., “database for fourth quarter financial results”). Inaddition, the description may include information identifying the hostcomputer (or other entity) that created the logical entity or otherinformation that specifies or assists in determining what to do with thelogical entity if found.

By assigning both an ELVID and a description, the table 141 may be usedto identify segments of data that may otherwise become “lost” in thecomputer system. For example, a logical volume (e.g., volume A of 126)may store a database. If the original host that created the logicalentity becomes unavailable (for example, with a catastrophic failure ofthat host), the failure could leave logical entities scattered about thestorage domain 121 with few mechanisms for determining what thoselogical entities are. Including a description column in the table 141permits identification or recovery of logical entities stored about thestorage domain 121, without the need to access the host which createdit.

This information would also permit the enterprise storage managementconsole 124 or other mechanism to inventory the data that is storedacross the storage domain. For example, enterprise storage managementconsole 125 or another mechanism could periodically verify that all ofthe logical entities stored in the storage domain are still being usedby one or more host computers. In this embodiment, the entity performingthe inventory could use the description field to determine not only whatthe logical object is generally, but also how to verify that it is stillin use (for example, which host computer or computers to query as towhether the data is still required). A variety of mechanisms may beemployed to perform this inventory process. For example, primary orother storage elements in the storage domain may determine or track whenit has been a long period of time since a logical entity had beenaccessed. Once the applicable threshold has been crossed, the storageelement could notify an enterprise storage management console thatstorage of that logical entity in the applicable storage element may nolonger be required.

ELVID to Physical Location Mapping.

FIG. 15A illustrates one embodiment of a method of using ELVIDs toaccess data in a computer system. At a step 150, a host specifies accessto a logical entity. This may be performed as a conventional step ofconventional computer systems where an application program (for example)on a host requires access to data. In step 150 of this particularembodiment, however, that access may be performed using an ELVID and alogical unit number (or information that may be mapped into an ELVIDwith or without a logical unit number).

Conceptually, an ELVID may be sufficient to specify a logical entity—aunique ELVID may be assigned to each logical entity in the storagedomain and may be used as such. (Under other embodiments, however, anELVID may specify a group of logical entities, with another fieldspecifying a subunit of the identified logical entity. For example, anELVID could specify a logical volume with a subfield dedicated tospecifying a file or hyper volume within the logical volume.)

In an embodiment described above, however, the ELVID is allocated thesame format (e.g., the same number of bits) as used by the host foridentifying a physical storage element (e.g., a worldwide name). If thisis the case, the host application and operating system software can usean ELVID in place of what would otherwise be a physical storage elementaddress. Allowing specification by the host computer of a logical unitnumber in addition to an ELVID to identify a logical entity allows ahost computer to see the same format as when the host computeroriginally accessed the data, without use of an ELVID—in some cases,reducing the amount of alteration to the host application or operatingsystem software necessary to implement and use ELVIDs.

FIG. 15B illustrates one example of accessing information according toone embodiment of the method illustrated in FIG. 15A. In FIG. 15B, ahost computer 156 is accessing a physical storage element 159 a, througha network 158. Initially, the host computer 156 specifies access to alogical entity. In many host computer systems, a configuration tablespecifies access to at least certain logical entities. In this example,a configuration table 157 a includes an entry for a logical entity“/DEV.” In conventional systems, the configuration table 157 a includesinformation specifying the physical location for that data. In thisexample, however, an ELVID plus a logical unit number is used in placeof what would otherwise specifically identify the physical storagelocation. In this manner and for this example, the configuration table157 a is used for the host to specify an ELVID plus logical unit number,as in step 150 of FIG. 15A.

Returning to FIG. 15A, it is determined whether the location of thespecified ELVID (plus logical unit number) is known, at a step 151.

Returning to FIG. 15B, this would correspond to examining a table 157 bto determine whether there is an entry identifying the location of thelogical entity.

Returning to FIG. 15A, if the location of the logical entity is notknown, then a mapping (i.e., determination of the physical location forthe ELVID) is found, at a step 153. This step may be performed byaccessing a centralized database that includes location of the ELVIDs,such as a database stored at the enterprise storage management console124 of FIG. 14. In other embodiments, some or all of the physicalstorage elements in the storage domain may be polled to determinewhether they have the appropriate logical entity.

Once the mapping of the ELVID has been found at step 153, the hostmapping table may be updated at a step 154. In the example of FIG. 15B,this would correspond to adding an entry in the table 157 b.

If the location of the ELVID is known, or once the location of the ELVIDhas been determined, the request can be translated to the physicallocation of the logical entity, at a step 152.

Referring to FIG. 15B, this may be done as a simple table lookup fromtable 157 b. That is, the ELVID or ELVID plus logical unit number can beconverted to an appropriate identifier for the physical location of thedata.

Returning to FIG. 15A, at a step 155, the data is accessed.

In the example of FIG. 15B, this may be performed by using the physicaladdress identified from the table 157 b to access the physical storagedevice 159 a, using network is 158. Using the example embodiments above,the physical storage element may be accessed using the appropriatenetwork address for the device together with a specification of thelogical entity stored on the device 159 b.

As described above, and in some embodiments, the particular operatingsystem used by a host may not accommodate a sufficient number of bitsfor an ELVID for a particular storage domain. For example, particularoperating systems may permit only a sufficient number of bits toidentify a logical unit, and fewer bits than the number necessary toidentify a worldwide name address on a fibre channel. In theseembodiments, and if the particular ELVID format chosen for the systemincludes the same number of bits as for a worldwide name, an additionallayer of mapping may be performed. That is, a host operating system mayspecify a logical entity in one format, followed by a conversion to anELVID together with a physical location.

FIG. 16 illustrates one embodiment of a method for accessing data insuch a system. At a step 161, the host specifies the logical entity tobe accessed.

At a step 162, the specified logical entity is mapped to an appropriateELVID and physical location. Thus, if the host entity specifies alogical unit number only, the operating system can map that logicalentity number to an ELVID or ELVID combined with a logical unit numberin the same manner that the operating system would otherwise map thespecified logical unit number to a physical address to access theprimary storage element over a network. In addition to mapping to theapplicable ELVID, the actual physical location is also identified.

In this example, a purpose of mapping to an ELVID rather than directlyto a physical location address is to permit the identification of theappropriate logical entity independent (as defined above) of physicallocation. Accordingly, if the logical entity has been moved, this may bedetected in the manner described below. In addition, changes to thephysical location can be implemented and updated more easily. The changein the physical location can be mapped to the ELVID. When the logicalentity with the applicable logical unit number is accessed, the ELVIDmay be used as a key to access the appropriate entry in the table andthe corresponding updated physical location is correctly accessed.

At a step 163, the data is accessed using the physical locationdetermined at step 162.

Referring again to FIG. 14, the second column of table 141 includes themapping location (or storage element location) for the logical entity.For the example of the logical entity volume A, an entry indicating itslocation—{PSE1_ID.VOL_A}—identifies the location of the storage forvolume A in the storage domain 121, in this example, the location beingwithin the primary-storage element 1 125 a.

In this example, the mapping information includes a separate identifier(“PSEI_ID”) of the storage entity in the storage domain 121 that storesthe logical entity. For example, if the primary storage element 125 a isa Symmetrix, the initial information in the mapping field of 141 a,PSEI_ID, is an identifier for that Symmetrix.

In one embodiment, the storage element identifier can be the networkaddress for the primary storage element 125 a. For example, in a fibrechannel network, the initial field {PSE1_ID} can be the appropriateworld wide name for primary storage element 1 125 a.

In the example of FIG. 14, the mapping information—“VOL_A”—thatidentifies the logical entity or storage location within the applicablestorage element. Thus, the entire mapping information {PSEI_ID.VOL_A}includes two fields, one corresponding to the particular storage element(e.g., a network address for that storage element) and one correspondingto a location within the storage element (e.g., a logical unitidentifier that may be used by the storage element to access theapplicable data).

In some embodiments, by including the mapping information, the hostapplication using the ELVID will have and use information that permitsit to directly access the appropriate storage element and data over thenetwork 123.

In the example of FIG. 14, the host 122 a also includes a table 140 thatshows a mapping of ELVIDs to storage areas. This would permit a storagemanagement application within host 122 a to directly access the primarystorage elements by converting the ELVID to the physical location forthe corresponding logical entity. In other embodiments, anothermechanism (such as an enterprise storage management console) may performthe function of mapping ELVIDs to physical locations for entities in thehost domain 120.

ELVID Checking.

In certain of the above embodiments, the application or file systemlevel software used an ELVID to identify logical as a part ofidentifying logical entities. The ELVID was then mapped to storagelocations for access to the data in the logical entity.

One alternative use of ELVIDs is for verification at the correct logicalentities being addressed at the physical location. Thus, if the physicallocation of a ELVID changes and a particular host has not had itsconfiguration table updated, the host will attempt to access theincorrect physical location. This can be detected using an ELVID,followed with an appropriate correction to the host that had attemptedto use the incorrect address.

FIGS. 17 and 18 illustrate one example of a method of using ELVIDs forverification of the location of the logical entity.

In FIG. 17, at a step 171, the host specifies access using a physicaladdress. This step may be performed in the conventional manner, usingthe mechanism the host would ordinarily use for addressing data.

At a step 172, however, the ELVID is added to the access request.Referring to FIG. 18, the host 122 a may use table 181 to identify anELVID with a particular physical address identifier specified in thedata access request by the host.

At a step 173 of FIG. 17, the physical address and the ELVID are matchedto assure that the host is accessing the correct data. In the embodimentof FIG. 18, this step may be performed by the enterprise storagemanagement console 124 using its ELVID table 180. In this embodiment,the host may be required to check with the enterprise storage managementconsole before accessing a logical object or volume (e.g., beforebeginning a session with a volume or before each access to a volume). Inan alternative embodiment, a similar table 182 may be stored at theprimary storage element 126. For simplicity, an example of one way ofperforming the step identified in 173 is described with reference to useof the table 182 on primary storage element 125 a.

For this embodiment, the host 122 a would specify a data access requestusing the physical ID, with the ELVID added from table 181 (as describedwith respect to 171 of FIG. 17). This request is then transmittedthrough network 123 to the primary storage element 125 a.

Receiving the data access request, the primary storage element 125 a maythen take the specified location and determine the ELVID at the addressspecified by the host 122 a.

If the ELVID and physical address match at the primary storage element125 a, the appropriate logical entity has been addressed by the host 122a. As a result, at a step 175 of FIG. 17, the data access may befulfilled. This should be the ordinary course of operation.

In some circumstances, however, the ELVID and the physical address maynot match at step 173. This may occur, for example, if the volume 126 isinitially stored in primary storage element 125 a and subsequentlymoved. If the configuration table 181 host 122 a has not been updated asa result of that move, the host 122 a may send (incorrectly) accessrequests to a stale data location.

If the ELVID and physical address do not match at step 173, thecorresponding host table (181 in the example described with respect toFIG. 18) must be updated.

Step 174 may be performed in a variety of ways. For example, whenever alogical entity is moved from a primary storage element, updatinginformation (e.g., a forwarding address) can be retained at the primarystorage element. In this case, the primary storage element could simplyforward the appropriate information to the host 122 a, which could thenupdate its configuration table 181. In an alternative embodiment, againin the context of FIG. 18, the primary storage element 125 a couldsimply notify the host 122 a that the ELVID verification process didcheck out. In this case, host 122 a could then obtain the appropriatephysical location for the ELVID from the enterprise storage managementconsole 124, which includes that information in table 180.

Moving Logical Entities

FIG. 19 illustrates one embodiment of a method for moving a logicalentity in a system that employs ELVIDs. At a step 191, a copy of thelogical entity is created. The copy may be created at the destinationlocation for the logical entity.

At a step 192, the appropriate ELVID tables are updated. These tableswere described generally above, and specify the appropriate physicallocation for the logical entity being moved.

At a step 193, the old memory is released.

The method of FIG. 19 is used most simply if all reading and writing ofthe logical entity can be frozen during the moving process. In thisembodiment, when the copy of the logical entity is created in step 191,there is no fear of updates being made to the logical entity at the oldlocation that would not be reflected where the copy is being created.Similarly, if accessing of the data is frozen (or at least writing ofthe date is frozen), there is no fear that updating of the ELVID willresult in the loss of data. For is example, if one host is writing to anold storage element because the applicable ELVID has not been updatedwhile another host is reading from a different location, the incorrectversion of the data may be used. Similarly, if only one host may accessthe logical entity and the ELVID update can be synchronized with theinput/output stream, a logical entity can be moved according to FIG. 19,without having to freeze access to the logical entity.

FIG. 20 illustrates one embodiment that permits movement of a logicalentity or object in the storage domain while host applications may stillbe accessing the logical object.

At a step 201, a synchronous mirror of the logical object isestablished.

FIG. 21A shows an example of a system with two host computers 210, 211and storage elements 212, 213. All reads and writes are performed for aparticular logical entity stored on a storage element 212. A mirror isestablished from the storage element 212 to the destination storageelement 213. In the example of FIG. 21A, the storage element 212establishes as mirror on storage element 213 using a remote datafacility, as described above. Naturally, other mechanisms could be usedto create a mirror.

For the particular mirror used in the example embodiment of FIGS.21A-21C, any writes that are performed on the primary copy (initially onstorage element 212) are automatically and immediately mirrored on amirror copy on storage element 215.

Returning to FIG. 20, at a step 202, all reads by the host computers aremoved to the new, destination storage element.

Referring to FIG. 21B, the state of the system of FIG. 21A is shown, butafter performing step 202. In particular, all reads to the applicablelogical entity from host computers 210, 211 are performed on the newstorage element 213. All writes, however, continue to be performed onthe initial storage element 212. These writes are mirrored, however, onthe new destination storage element 213.

By moving the reads before the writes, coherence of the data can beassured when more than one host is accessing a logical entity. All ofthe writes are maintained on what will become the old copy 212 until allreads by all hosts in the storage domain has been moved. Although readsare occurring on the new copy and writes are made on the old copy, eachwrite is synchronously mirrored onto the new copy.

Returning to FIG. 20, at a step 203, all of the writes are moved to thenew storage element. This is illustrated in FIG. 21C, where all readsand writes are being performed on primary storage element 213.

To move writes to the new location, in this embodiment, the storageelement 212 with the old copy refuses to acknowledge any reads to thatstorage element (since reads have been moved, none should occur). Themechanisms which map writes are then updated map writes to the newlocation.

The mechanism used to move reads and writes can differ depending on theparticular mechanism for using ELVIDs to access data. In an embodimentwhere ELVID checking is performed, a storage element 212 can simplyrespond to read requests with a notification that the address has moved,to accomplish moving of reads from the original copy of storage element212 to the new copy of storage element of 213. The same could be done ata later point in time to move the writes. (The same mechanism can beused to move reads and writes simultaneously in systems where this isdesired.)

Using ELVID mapping at the host computer as described above, ELVIDchecking could also be implemented at the storage elements to supportmovement of logical entities from one storage element to another as wellas used as an additional check to assure that the appropriate entity isbeing accessed. In the alternative, the enterprise and storage networkcontroller 124 of FIG. 14 (or some other entity in the host or storagedomain) could track and update which host computers have been notifiedof the identity of a location of a ELVID and simply issue appropriatecontrol messages to update the host configuration tables on those hosts.In these embodiments, the configuration table 141 may include trackinginformation as to which hosts in the host domain 120 have been notifiedof the location of a logical entity.

In some embodiments, one of the entities in the system can track andmaintain a log of the operations performed during a move. Where anenterprise storage management console is provided, for example, theenterprise storage management console can track and log the movementprocess for archival purposes and, in the case of failure, to permitrecovery of the logical entity.

A similar mechanism can be used to restore a backup copy of a logicalentity, to rollback the system to an earlier state or to recover fromloss of data.

FIG. 22, illustrates a block diagram of one method for performing this.At a step 221, the backup copy of the logical entity to be restored isidentified.

At a step 222, the appropriate ELVID tables are updated. As describedabove, the particular mechanism for updating the ELVID tables willdepend upon the manner in which ELVID tables are used. Two examples ofthe use of ELVIDs are described above ELVID mapping and ELVID checkingsystems. Depending on the manner of the use of ELVIDs, the ELVID tablesare updated appropriately also as described above.

Access Checking

In the embodiments described above, as well as other embodiments, ELVIDsmay be used to restrict access to logical entities.

One level of restricting access to ELVIDs may be described withreference to FIG. 15A. In this embodiment, when a host first accesses alogical entity, the location for the logical entity is not known at step151. Accordingly, when the host seeks to find the location of thelogical entity at step 153, an access authorization or verificationprocess may occur. For example, if the enterprise storage managementconsole 124 of FIG. 14 is used to notify the host of the ELVID mapping,the enterprise storage management console can first verify that accessis authorized.

In addition or in the alternative, access verification can occur at thestorage element. Thus, when a storage element receives an access requestwith an ELVID at step 173 of FIG. 17, the storage element can verifythat the access request from this particular host or user is authorized.(For example, by querying the enterprise management console the firsttime a host or user releases a logical entity and by maintaining acorresponding entry in a configuration table if access is authorized.)If not, the storage element can notify both the host and the enterprisestorage management console that the access request has been denied.

Access checking can be used not only to verify that a user can accessthe logical entity but also can be used to permit some users to read,but not write, a logical entity.

To assist access checking in the embodiment of FIG. 18, the variousELVID mapping tables 180, 181 and 182 may include access information.This may include a column in the table that specifies, for example,unrestricted access or access only to specified hosts or users. Thevarious schemes for determining access to logical entities would beapparent to one of ordinary skill in the art based on the disclosureprovided herein. For example, the host that originally creates a logicalentity may specify the access restrictions (if any) for a logicalentity. Updates or modifications to the access list can be tracked.

Example System FIG. 23 illustrates one embodiment of a computer systemthat uses ELVIDs for a variety of functions. In this particularembodiment, many of the ELVID functions are performed by a storagemanagement controller 232. This storage management controller may be aseparate computer system, including its own processor and memory. Thestorage management controller 232 is connected to a network 234. Thecomputer network 234 handles communications among the host computers(e.g., host computer 230) and storage elements (e.g., primary storageelement 235).

The host computer 230 performs application processing and may supportone or more users.

The primary storage element 235 is a storage element in the computersystem.

The primary storage element may be any computer storage medium,including the Symmetrix storage element described above.

Although the network 234 is illustrated as a single network, it mayinclude one or more subnetworks. For example, the storage managementcontroller 232 may communicate with one or more of the host computers(e.g., 230) or storage elements (e.g., 235) using a separate network,such as a TCP/IP network. The host computer 230 and primary storageelement 235 may communicate over that network or may instead use adifferent network, such as a fibre channel.

In the example of FIG. 23, the storage management controller 232includes a variety of software modules for implementing variousfunctions related to ELVIDs. (“Module” as used herein refers genericallyto a mechanism to perform a function. Thus, a “module” may beimplemented as a software module or may be implemented in firmware,hardware of a combination of these.) Of course, the storage managementcontroller may include a number of other software or other modules thatperform additional storage management functions, such as maintaining orcoordinating primary verses secondary storage, as generally describedabove.

In the example of FIG. 23, the storage management controller 232includes an ELVID assignment module 232 a. The ELVID assignment moduleis responsible for assigning ELVIDs to logical entities, when requestedby a host computer (e.g., host 230). As described generally above, theELVID should be unique to the applicable logical entity.

The ELVID assignment module coordinates with an ELVID database manager233 a, which controls access to an ELVID database 233 b. The ELVIDdatabase may be a table or other database, as generally described above.Thus, in an example embodiment, the ELVID database may includeinformation specifying an ELVID, its location among the storageelements, e.g., storage element 235 and description field or fieldsincluding information of the types described above.

In the example embodiment of FIG. 23, the storage management controller232 also includes an inventory module 232 b. The inventory module mayperform inventorying functions, as described above. For example, theinventory module may periodically query the applicable host computer 230to determine whether the logical entity is still required for use. Inaddition, the inventory module may periodically determine whether thelogical entity may be moved from primary storage to secondary storage.This may be performed by, for example, polling the host computers thataccess a logical entity to determine whether the logical entity is stillin use.

The storage management controller 232 of FIG. 23 also includes an accessmanager 232 c. The access manager may receive access requests from thehost computers, e.g., 230. The access manager would be responsible forsending physical location information to any host computer that requestsaccess to the logical entity. The access manager 232 c may also performverification that any host or user on the host requesting access to alogical entity is permitted that access, all as described above.

The entity movement module 232 d would be responsible for managing themovement of logical entities within or among storage elements in thesystem. The entity movement module may, for example, employ the methodsdescribed above for moving an entity between storage elements.

Finally, the storage management controller 232 includes an interface 237to coordinate communication among the other components of the computersystem.

In the example of FIG. 23, the host computer 230 includes an ELVIDinterface module 231 a and an ELVID database 231 b, the ELVID databasemay include a table as generally described above. The ELVID interfacemodule 231 a may perform the functions described above for managing useof the ELVIDs. The ELVID interface module 231 a may interface with theoperating system on the host computer 230, for example interceptingrequests for access to storage and mapping or adding ELVID informationto requests as appropriate.

The primary storage element 235, in the example of FIG. 23, includes anELVID verification module 236 a and an ELVID database 236 b. Again, theELVID database may include the information described above with respectto configuration tables for storage elements. The ELVID verificationmodule 236 a may perform a function of checking ELVIDs against physicallocations to assure that any access request is receiving data for thecorrect logical entity.

For each of the host 230, storage management controller 232 and primarystorage element 235, additional software and hardware components may beincluded to perform the other functions in the computer system. Inaddition, the example illustrated in FIG. 23 is illustrative—there aremany ways that the functions described above may be implemented.

The various methods above may be implemented as software on a floppydisk, compact disk, or other storage device, for use in programming orcontrolling a computer. The computer may be a general purpose computersuch as a work station, main frame or personal computer, that performsthe steps of the disclosed processes or implements equivalents to thedisclosed block diagrams and may include in addition, or instead,special purpose hardware for performing some or all of such steps orsimilar steps. The software may be included on a diskette as a completesystem or as enhancements to an existing system, permitting the systemto perform the methods described herein.

Having thus described at least illustrative embodiments of theinvention, various modifications and improvements will readily occur tothose skilled in the art and are intended to be within the scope of theinvention. Accordingly, the foregoing description is by way of exampleonly and is not intended as limiting. The invention is limited only asdefined in the following claims and the equivalents thereto.

What is claimed is:
 1. A method of specifying a logical volume of aplurality of logical volumes in a networked computer system including aplurality of storage elements, a plurality of host computers, and acommunications network that couples together the plurality of storageelements and the plurality of host computers, wherein each of theplurality of storage elements is a separate unitary storage systemcomprising at least one physical storage device, wherein at least one ofthe plurality of logical volumes is stored on each of the plurality ofstorage elements, the method comprising acts of: assigning a logicalvolume identifier to the logical volume, the logical volume identifierbeing usable by at least one of the plurality of host computers toaccess, via the communications network, the logical volume on at leastone of the plurality of storage elements; assuring that the logicalvolume identifier uniquely identifies the logical volume among theplurality of logical volumes stored in the networked computer system;and maintaining the same logical volume identifier when the contents ofthe logical volume have been electronically moved to a new storagelocation in the networked computer system.
 2. The method of claim 1,wherein each of a plurality of users is capable of accessing the logicalvolume, and further comprising an act of: providing each user capable ofaccessing the logical volume with the logical volume identifier.
 3. Themethod of claim 1, wherein the act of assigning comprises an act ofassigning the logical volume identifier, independent of a location ofstorage for the logical volume.
 4. The method of claim 1, wherein theact of assigning comprises an act of assigning the logical volumeidentifier to a partition.
 5. The method of claim 1, wherein the act ofassigning comprises an act of assigning the logical volume identifier toa hyper-volume.
 6. The method of claim 1, wherein the act of assigningcomprises an act of assigning the logical volume identifier to a stripedvolume.
 7. The method of claim 1, wherein the act of assigning comprisesan act of assigning a unique logical volume identifier to each of aplurality of logical volumes.
 8. The method of claim 7, furthercomprising an act of maintaining a database identifying the logicalvolume identifiers and corresponding locations.
 9. The method of claim8, wherein the act of maintaining comprises an act of maintaining thedatabase at a storage management controller.
 10. The method of claim 7,wherein the database further includes descriptive information for eachcorresponding logical volume.
 11. The method of claim 10, wherein thedescriptive information comprises information identifying each hostcomputer that has accessed each corresponding logical volume.
 12. Themethod of claim 10, wherein the descriptive information comprisesinformation identifying each host computer authorized to access thecorresponding logical volume.
 13. The method of claim 10, wherein thedescriptive information comprises information identifying acorresponding host that created the corresponding logical volume. 14.The method of claim 10, wherein the descriptive information comprisesinformation identifying access history for a corresponding logicalvolume.
 15. The method of claim 10, wherein the descriptive informationcomprises information describing the nature of data stored in thelogical volume.
 16. The method of claim 1, wherein the logical volumeidentifier has a format corresponding to an address format used formemory access requests made by one of the host computers to one of thestorage elements.
 17. The method of claim 16, wherein the logical volumeidentifier has a format corresponding to a format of a network addressfor accessing one of the storage elements.
 18. The method of claim 17,wherein the logical volume identifier corresponds to a format of a worldwide name.
 19. The method of claim 16, wherein the logical volumeidentifier has a format corresponding to a format of a network addressfor one of the storage elements and a logical unit identifier.
 20. Themethod of claim 16, wherein the logical volume identifier has a formatcorresponding to a format of a world wide name and a logical unitidentifier.
 21. A method of specifying a logical volume of a pluralityof logical volumes in a networked computer system including a pluralityof storage elements, a plurality of host computers, and a communicationsnetwork that couples together the plurality of storage elements and theplurality of host computers, wherein each of the plurality of storageelements is a unitary storage system comprising at least one physicalstorage device, wherein at least one of the plurality of logical volumesis stored on each of the plurality of storage elements, the methodcomprising acts of: assigning a logical volume identifier to the logicalvolume, the logical volume identifier being assigned independent of theidentity of the host computer initially creating the logical volume andbeing usable by at least one of the plurality host computers to access,via the communications network, the logical volume on at least one ofthe plurality of storage elements; and assuring that the logical volumeidentifier uniquely identifies the logical volume among the plurality oflogical volumes in the networked computer system.
 22. A method ofspecifying a logical volume of a plurality of logical volumes in acomputer system including a plurality of storage elements and aplurality of host computers, wherein each of the plurality of storageelements is a unitary storage system comprising at least one physicalstorage device, wherein at least one of the plurality of logical volumesis stored on each of the plurality of storage elements, the methodcomprising acts of: assigning a logical volume identifier to the logicalvolume, the logical volume identifier having a format corresponding toan address format used for memory access requests made by one of thehost computers to one of the storage elements; and assuring that thelogical volume identifier uniquely identifies the logical volume amongthe plurality of logical volumes stored on the plurality of storageelements.
 23. The method of claim 22, wherein the logical volumeidentifier has a format corresponding to a format of a network addressfor one of the storage elements.
 24. The method of claim 23, wherein thelogical volume identifier corresponds to a format of a world wide name.25. The method of claim 23, wherein the logical volume identifier has aformat corresponding to a format of a network address for one of thestorage elements and a logical unit identifier.
 26. The method of claim22, wherein the logical volume identifier has a format corresponding toa format of a network address for one of the storage elements and alogical unit identifier.
 27. A method of storing and maintaining alogical volume of a plurality of logical volumes on a networked computersystem having a plurality of storage elements, a plurality of hostcomputers, and a communications network that couples together theplurality of storage elements and the plurality of host computers,wherein each of the plurality of storage elements is a unitary storagesystem comprising at least one physical storage device, wherein at leastone of the plurality of logical volumes is stored on each of theplurality of storage elements, the method comprising acts of:identifying the logical volume; assigning an enterprise logical volumeidentifier (ELVID) to the logical volume, the ELVID uniquely identifyingthe logical volume among the plurality of logical volumes and beingusable by at least one of the plurality of host computers to access thelogical volume over the communications network; allocating memory on atleast one storage device; saving the logical volume on the allocatedmemory; and maintaining the same ELVID when the contents of the logicalvolume have been electronically moved to a new storage location in thestorage elements.
 28. The method of claim 27, wherein the ELVID isindependent of a physical location of the allocated memory.
 29. Themethod of claim 27, further comprising a step of recording the ELVID andinformation identifying a physical location of the allocated memory in adatabase.
 30. The method of claim 29, further comprising a step ofstoring the database at a storage management console.
 31. The method ofclaim 29, further comprising a step of storing the database at a hostcomputer that generated the logical volume.
 32. The method of claim 27,wherein the logical volume is created by a first host computer; and themethod further comprises a step of notifying a second host computer ofthe ELVID to permit access to the logical volume.
 33. A computer system,comprising: a plurality of storage elements including a plurality oflogical volumes stored thereon, each of the plurality of storageelements being a unitary storage system comprising at least one physicalstorage device, wherein at least one of the plurality of logical volumesis stored on each of the plurality of storage elements; means forassigning a logical volume identifier to a logical volume of theplurality of logical volumes for storage on the storage elements, thelogical volume identifier being usable by at least one computer remoteto the computer system to access the logical volume on at least one ofthe plurality of storage elements; and means for assuring that thelogical volume identifier uniquely identifies the logical volume amongthe plurality of logical volumes stored on the plurality of storageelements and for maintaining the same logical volume identifier when thecontents of the logical volume have been electronically moved to a newstorage location in the storage elements.
 34. The computer system ofclaim 33, further comprising: means for providing each of a plurality ofhost computers capable of accessing the logical volume with the logicalvolume identifier.
 35. The computer system of claim 33, wherein themeans for assigning comprises means for assigning the logical volumeidentifier, independent of a location of storage for the logical volume.36. The computer system of claim 33, wherein the means for assigningcomprises means for assigning a unique logical volume identifier to eachof a plurality of logical volumes.
 37. The computer system of claim 36,further comprising: means for maintaining a database identifying theunique logical volume identifiers and corresponding locations.
 38. Thecomputer system of claim 37, wherein the means for maintaining comprisesa step of maintaining the database at a storage management controller.39. The computer system of claim 36, wherein the database furtherincludes descriptive information for each corresponding logical volume.40. The computer system of claim 39, wherein the descriptive informationcomprises information identifying each host computer that has accessedeach corresponding logical volume.
 41. The computer system of claim 39,wherein the descriptive information comprises information identifyingeach host computer authorized to access the corresponding logicalvolume.
 42. The computer system of claim 39, wherein the descriptiveinformation comprises information identifying a corresponding host thatcreated the corresponding logical volume.
 43. The computer system ofclaim 39, wherein the descriptive information comprises informationidentifying access history for a corresponding logical volume.
 44. Thecomputer system of claim 39, wherein the descriptive informationcomprises information describing the nature of data stored in thelogical volume.
 45. The computer system of claim 33, wherein the logicalvolume identifier has a format corresponding to an address format usedfor memory access requests made by one of the host computers to one ofthe storage elements.
 46. The computer system of claim 45, wherein thelogical volume identifier has a format corresponding to a format of anetwork address for one of the storage elements.
 47. The computer systemof claim 46, wherein the logical volume identifier corresponds to aformat of a world wide name.
 48. The computer system of claim 46,wherein the logical volume identifier has a format corresponding to aformat of a network address for one of the storage elements and alogical unit identifier.
 49. The computer system of claim 45, whereinthe logical volume identifier has a format corresponding to a format ofa network address for one of the storage elements and a logical unitidentifier.
 50. A computer storage system, comprising: a plurality ofstorage elements configured to store thereon a plurality of logicalvolumes, wherein each of the plurality of storage elements is a unitarystorage system comprising at least one physical storage device, whereinat least one of the plurality of logical volumes is stored on each ofthe plurality of storage elements; and means for assigning an enterpriselogical volume identifier (ELVID) to a logical volume of the pluralityof logical volumes stored, or to be stored, on at least one of theplurality of storage elements, the ELVID uniquely identifying thelogical volume among the plurality of logical volumes and being usableover a communications network by at least one computer to access thelogical volume on the at least one of the plurality of storage elements.51. The computer storage system of claim 50, further comprising: meansfor allocating memory on the plurality of storage elements; and meansfor saving the logical volume on the allocated memory.
 52. The computerstorage system of claim 50, wherein the means for assigning comprisesmeans for assigning the ELVID independent of a physical location of theallocated memory.
 53. The computer storage system of claim 50, furthercomprising means for recording the ELVID and information identifying aphysical location of the allocated memory in a database.
 54. Thecomputer storage system of claim 50, further comprising: a storagemanagement controller; and wherein the means for recording comprisesmeans for storing the ELVID and identifying information in a databasecoupled to the storage management controller.
 55. The computer storagesystem of claim 54, further comprising means for storing the database ata host computer that generated the logical volume.
 56. The computerstorage system of claim 50, wherein the logical volume is created by afirst host computer, and the computer storage system further comprises:means for notifying a second host computer of the ELVID to permit accessto the logical volume.
 57. The computer storage system of claim 50,wherein the ELVID has a format corresponding to an address format usedfor memory access requests made by one of the host computers to one ofthe plurality of storage elements.
 58. A storage management controllerfor a computer storage system that includes a plurality of storageelements, wherein each of the plurality of storage elements is a unitarystorage system comprising at least one physical storage device, thestorage management controller comprising: an enterprise logical volumeidentifier (ELVID) assignor to assign ELVIDs to a plurality of logicalvolumes to be stored on the computer storage system, wherein an ELVIDassigned to a first logical volume of the plurality of logical volumesuniquely identifies the first logical volume among the plurality oflogical volumes, wherein at least one of the plurality of logicalvolumes is to be stored on each of the plurality of storage elements;and an interface to permit communication with the plurality of storageelements, wherein the ELVID is usable by at least one computer remote tothe plurality of storage elements to access the first logical volumethrough the interface.
 59. The storage management controller of claim58, further comprising: an ELVID database manager.
 60. The storagemanagement controller of claim 58, further comprising: an ELVID accessmanager to control access to logical volumes identified by ELVIDs. 61.The storage management controller of claim 58, further comprising: alogical volume movement manager.
 62. The storage management controllerof claim 58, wherein ELVIDs have a format corresponding to a format of anetwork address for one of the plurality of storage elements.
 63. Thestorage management controller of claim 58, wherein ELVIDs correspond toa format of a world wide name.
 64. The storage management controller ofclaim 58, wherein ELVIDs have a format corresponding to a format of anetwork address for one of the plurality of storage elements and alogical unit identifier.